Aiming at the characteristics of small size and high data dimensionality of speech emotion datasets， to solve the problem of long-range dependence disappearance in traditional Recurrent Neural Network （RNN） and insufficient excavation of potential relationship between frames within the input sequence because of focus on local information of Convolutional Neural Network （CNN）， a new neural network MAH-SVM based on Multi-Head Attention （MHA） and Support Vector Machine （SVM） was proposed for Speech Emotion Recognition （SER）. First， the original audio data were input into the MHA network to train the parameters of MHA and obtain the classification results of MHA. Then， the same original audio data were input into the pre-trained MHA again for feature extraction. Finally， these obtained features were fed into SVM after the fully connected layer to obtain classification results of MHA-SVM. After fully evaluating the effect of the heads and layers in the MHA module on the experimental results， it was found that MHA-SVM achieved the highest recognition accuracy of 69.6% on IEMOCAP dataset. Experimental results indicate that the end-to-end model based on MHA mechanism is more suitable for SER tasks compared with models based on RNN and CNN.